Towards Multimodal Perception and Semantic Understanding in a Developmental Model of Speech Acquisition

Babbling is a crucial process in young infants for acquiring articulatory control. By constantly trying out motor commands and observing the consequences that they cause in the environment, they develop an understanding of their own body and learn to control their articulators in order to produce meaningful speech. In earlier works, we proposed a developmental model of speech acquisition that learns to control a 3-d vocal tract simulation for producing vowel or syllable sounds. The system self-organizes learning according to speech it perceives from its environment, so that ambient speech shapes the learning process. Here, we discuss how the proposed model could be extended to form a bridge between perception, on the one end, and semantics, on the other end. The idea is to connect acoustic perception with other perceptual modalities. As an example, we discuss how the system could integrate visual input in its learning loop. By learning associations between acoustic targets and simultaneous visual perceptions, such an enhanced model could produce speech not only in reaction to acoustic input, but also triggered by visual input. Vision, thus, could help to establish a common ground between learner and tutor for interactive articulatory learning. I. A DEVELOPMENTAL MODEL OF SPEECH ACQUISITION Infants explore in a goal-directed manner from the very beginning. Studies show that even neonates orient themselves towards more interesting targets [1] and prenatal exposure to their native language seems to influence infants’ early babbling behavior (e.g. [2]). We combined these ideas in a developmental model of speech acquisition in which we model the influence of ambient speech on the learning process [3]. We provide a set of speech sounds to the system from which it extracts the important components from a high-dimensional acoustic space representation via linear discriminant analysis. The resulting 2-d goal space forms a low-dimensional representation of speech that the system is exposed to in its environment. Through this dimension reduction, full syllables are projected onto a single point in goal space. Fig. 1 depicts such a goal space trained from ambient speech sound sets consisting of the three syllables /a/, /ba/ and /ma/. This research has been supported by the Cluster of Excellence Cognitive Interaction Technology ’CITEC’ (EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG), and is related to the European Project CODEFROR (FP7-PIRSES-2013-612555) ́1 0 1 ́1 ́0.5 0 0.5 1 a